feat(hub): dependents knobs, frontier_extend step, Resources panel + poll-resilience fixes#51
Merged
Merged
Conversation
feat: integrate all services
release: prepare v1.0.0 (PyPI rename + Trusted Publisher)
ci(release): add release-please for automated semver releases
release(docs): landing redesign + Docusaurus theming + nodes registry
…end step Quest builder now emits crawl_dependents / crawl_dependencies / min_stars / max_dependents from the form, with a max_dependents=0 sentinel mapped to null (no cap). Crawler timeout also scales with max_rounds (1800s per round) so deeper crawls don't hit the prior flat 30-min ceiling. New pipeline step `frontier_extend`: reads an existing crawler-graph.json, finds the dependents listed but never explored (the BFS frontier), submits a fresh crawl seeded with those nodes, and merges the result back over the canonical path. Cheaper than re-running with max_rounds++. Disabled by default; enable via the hub form or in the quest YAML. The hub form exposes the new step plus GET /api/pipeline/frontier-preview which reports frontier_size and a sample of seeds before launch.
Polling get_status / get_extract_job under load was failing the whole pipeline step on a single ReadTimeout or Connection-reset, even though the job on the other side was still alive and would complete normally. Both wait_for_completion / wait_for_extract now swallow up to 5 consecutive httpx (or wrapped RuntimeError, for the extractor) poll failures with a warning each, escalating only when the budget is exhausted. Real failure modes (status=failed, status=cancelled, deadline exceeded) still propagate immediately. Crawler HTTP client timeout bumped 30s -> 60s while we're here.
GET /api/admin/resources returns disk usage per mount, MemTotal + MemAvailable, /proc/loadavg, and `docker system df` totals (images, containers, volumes with reclaimable bytes). 30s server-side cache. The /admin page polls every 15 min by default and colours rows yellow at 75% (disk) / 80% (mem) / 1.0 load-per-core, red at 90% / 92% / 2.0. Stdlib + Docker SDK only; no psutil added to the image. Surfaces the "we are close to the wall" signal early — we crossed 95% disk usage twice this week before noticing.
caviri
added a commit
that referenced
this pull request
May 18, 2026
Conflict: src/open_pulse/gui/hub/main.py
Both branches added new routers / imports to the FastAPI app:
· develop → ``admin`` router (the /admin Resources panel from
PR #51, commit 915151f)
· this branch → ``hub`` + ``chaoss_routes`` routers + the
_propagate_globals helper that mirrors the shared
template env to instances with their own filters
Resolved by keeping all three. New combined wiring:
app.include_router(crawler.router)
app.include_router(admin.router) # develop
app.include_router(hub.router) # ours
app.include_router(hub.api) # ours
app.include_router(chaoss_routes.router) # ours
base.html merged cleanly (the new ``/admin`` nav entry from develop
sits alongside the new ``/chaoss`` nav entry we already added).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three commits, each independently revertable.
Hub form: bidirectional dependency knobs + frontier_extend
crawl_dependents,crawl_dependencies,min_stars,max_dependents(0 → no cap).max_rounds(1800s/round), unblocking depth ≥ 3 runs that were hitting the prior flat 30-min ceiling.frontier_extend: re-seeds the crawler with the dependents listed in an existing graph but never explored, then merges the result back. Cheaper than re-crawling withmax_rounds++. Off by default.GET /api/pipeline/frontier-previewreportsfrontier_size+ a sample before launching.Services: tolerant polling
wait_for_completion(crawler) andwait_for_extract(metadata extractor) now swallow up to 5 consecutive transient HTTP errors with a warning each, instead of failing the whole step on a singleReadTimeout/Connection reset. Real terminal states (failed,cancelled, deadline) still propagate immediately./admin Resources panel
GET /api/admin/resourcesreturns disk usage per mount,MemTotal+MemAvailable,/proc/loadavg, anddocker system dftotals (with reclaimable bytes per resource class)./adminpage polls every 15 min and colours rows yellow at 75/80% / 1.0-per-core, red at 90/92% / 2.0. Stdlib + Docker SDK only; nopsutildependency added.Test plan
frontier_extendenabled/disabled.frontier_extendagainst the live graph: 230 frontier nodes → merged → 422 dependents, frontier shrunk to 15.failed→ still raises./adminpage renders;/api/admin/resourcesreturns disk/mem/cpu/docker on the live host.Out of scope (filed separately)
open-pulse-crawler: cancelled jobs cannot expose their partial graph via/api/v1/graph. Cost us 2,577 explored repos on one run. Report drafted for the crawler team.